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FINAL REPORT 


PANEL 9 

safety/reliability AND QUALITY ASSURANCE 


I. INTRODUCTION 


a. Scope 

The Safety/R&QA Panel 9 to the MSC Apollo 13 Investigation Team 
was established to provide the fQllowing functions : 

1. Act in an advisory capacity to the Team Manager 

2. Provide specific documentation and documentation control 

3* Provide technical support to Panels 1, 6, 7 , 8, 11 and 

members of the Investigation Board 

This report summarizes these activities and presents Safety/R&QA (Panel 9 ) 
recommendations . 

b. Organization 

The Safety/R&QA support through Panel 9 and through Panels 1, 5, 

6, 7 , 8, and 11 is shown on the preceding page. Names of key people 
supporting these panels are listed. These people were supported by 
personnel from NASA Safety/R&QA elements, from their Boein ; Company and 
General Electric Company support contractors, and by apprc uriate hardware 
contractor personnel. 

The Safety/R&QA summary panel reports referenced in Section VI provide 
additional details of the support supplied. It is estimated that at the 
peak of the activity, the Safety/R&QA Offices had more than 200 personnel 
directly supporting the investigation. 

In addition to the technical support referenced above , ap‘ roximately two 
thousand documents were retrieved, reproduced or otherwis made available 
to the panels and to members of the Investigation Board. Most of the 
documents are listed in the individual panel summary reports referenced in 
Section VI of this report. Others supplied to members of the Investigation 
Board are listed in Enclosure A. Still others supplied to team panels and 
not otherwise referenced are listed in Enclosure B of this report. 
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c. Activities Trior to Formation of the Apollo 13 Investigation Team 


During the Apollo 13 mission, the planned Safety/R&QA support 
was set up to review flight anomalies as they occurred and to provide 
related data, Information, and recommendations through the Safety/R&QA 
representatives on the Mission Monitoring Team. At the time of the cryo- 
genic tank incident, the Safety/R&QA organization furnished supporting 
data and information related to the Incident and reviewed changes in flight 
plan and flight procedures as they related to hardware capabilities to 
assure "safe recovery". Data, information, and recommendations from these 
activities were furnished in real-time to the Mission Monitoring Team. 
Immediately after splashdown an effort was initiated to collect data on 
certification and problem history and KSC checkout activities on problem 
related hardware. In addition, timelines and other basic information were 
studied in preparation for the formal team activities. Safety/RffeQA support 
continued as the Apollo 13 Investigation Team was being organized. Ihe 
remainder of this report describes the support furnished to the Investigation 
Team. 


II. GENERAL ACTIVITIES BY SAIETO/R&QA 


Panel 9 supported the investigation in the following areas: 

. Served as overall Safety/R&QA advisor to the team 
Provided support to board members 
. Provided support to Panels 1, 5, 6, J, 8, and 11 as follows: 

PANEL 1, Spacecraft Incident Investigation 

Compared previous safety related flig.it and test anomalies 
with Apollo 13 flight data 

• Reviewed previous safety assessments of Apollo 7 through 13 

* Evaluated test results and test analyses 

* Provided Data Centers to enable: 

• Control of pertinent historical documents, records, and data 
Security of data 

• Reliability/Quality Records/Evaluation 

* Prepared history and reevaluated certification test data 
for cryogenic 0£ tank 


mm 


r 

r 


• Prepared synopsis of test procedures used at Beech Aircraft 
Corporation on the cryogenic O 2 tank number 2 and all 
associated equipment 

♦ Provided failure history of electrical connectors used on 
the cryogenic O 2 tank 

PANEL 5 j Corrective Action Study and Implementation 
. Analyzed safety impact of recommendations 

• Evaluated alternatives relative to previous safety assessments 
Evaluated test requirements and results 

Assisted in establishing qualification requirements for possible 
cryogenic O 2 tank redesign 

Evaluated alternative designs 

PANEL 6, Related Systems Evaluation 

Analyzed proposed revisions for reliability and safety considerations 

• Evaluated alternatives relative to previous safety assessments 

Assessed nonmetallic material compatibility of all oxygen, 
fuel, and oxidizer tank assemblies and line components with the 
exception of cryogenic oxygen tanks 

• Provided information on contamination requirements 

• Reviewed and assessed certification and failure history, failure 
mode and effects analysis and single point failure summaries 
and limited life requirements 

• Provided RSbQA inputs to Systems Engineering team members 
Provided inputs on emergency backup modes 

. Provided information on pressure vessel explosion protection 

• Provided information on pyro initiator debris 

• Provided circuit breaker data 

• Reviewed and provided inputs into final report 

• Determined energy levels of the effect of faulted electrical com- 
ponents oxygen and oxidizer interfaces with the exception of the 
cryogenic Og tank 
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PANEL 7> Reaction Processes in High Pressure Fluid Systems 




Furnished documentation relative to LOX compatibility of 
materials 

Listed and reevaluated two waivers against Apollo 13 on 0£ 
fluid cleanliness 

PANEL 8, High Pressure Oxygen Systems Survey 

Gathered and reviewed information on fires and/or explosions 
in P 2 enriched environments of greater than ambient pressures 

Assembled applicable design standards and safety criteria 

Furnished procedures used at MSC for cleaning of facility O? 
equipment 

PANEL 11, Administration, Communications, and Procurement 
• No significant activity 


III. FINDINGS 


1. The existing system for control of the hardware certification 
test programs and assessment of completion is adequate. 

2. Formal closeout and recurrence control of reported hardware 
problems is adequate. 

3. The current MSC method for R&QA and Engineering review of hardware 
nonconformances and material review board actions is adequate. 

1+. Adequate background information on both specific and related 
generic hardware is contained in the record system. 

5. There is a comprehensive spacecraft cabin materials control 
program. The area of materials compatibility in high pressure fluids 
is not completely understood. 
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IV. RECOMMENDATIONS 


1. It is recommended that the present management system involving 
decisions at all levels on flight hardware he reviewed. The objective 
would be to optimize visibility to management concerning background on 

the specific and related generic hardware during the decision-making process. 

2. It is recommended that a comprehensive testing program to establish 
materials compatibility with high pressure oxygen and other fluid systems 

as used in spacecraft systems be undertaken by MSC. 


V. SUMMARY 


The Safety/R&QA Offices concur with +hc findings and corrective action 
defined to date relative to the Apollo 13 cryogenic tank failure, although 
certification and acceptance test requirements of the new tanks have not 
yet been completely defined. 

Disposition of the recommendations submitted by Safety/R&QA representatives 
through their assigned panels has been satisfactory. 



M. L. Raines 

Chairman, Panel 9 

NEC Apollo 13 Investigation Team 
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VI. REFERENCES 


The following is a list of Summary Reports submitted by Safety/R&QA 
support to the Apollo 13 Investigation Team Ifcnels: 

Reference 1? Final Report, Panel 1, Group b. Historical Records 
Handling and Cataloging, ND/May 18, 1970. 

Reference 2: Panel 5^, CSM, Corrective Action Study and Implementa 

tion Panel, Safety and R&QA Report to Panel 9 t SN/May 15 , 1970. 

Reference 3? Panel 5b, Lunar Module, Corrective Action Study and 
Implementation Panel, R&QA and Safety Report to Ifenel 9> NA/May 18, 1970 

Reference 4: Panel 5c, Government -Furnished Equipment, Corrective 

Action Study and Implementation Panel, Safety and R&QA Report to Panel 9 
SN/May 15, 1970. 

Reference 5: Input to Panel 9, Safety and R&QA Summary of R&QA Par 

ticipation to Panel 6, Related Systems Evaluation, NB/70-M525, May l4, 
1970, and NB/70-M548, May 20, 1970. 

Reference 6: Panel 7> Reaction Processes in Flight Pressure Fluid 

Systems, ND/May 18, 1970. 

Reference 7: Panel 8, High Pressure Oxygen Systems Panel, Safety 

and R&QA Report to Panel 9> SM/May lU, 1970. 





ENCLOSURE A 


Panel 9 ~ Safety and R&QA Documents Transmitter to the 
Apollo 13 Investigation Board (Messrs. Whitten and Ginter) 


Documents listed below were transmitted to: 


a. Mr. D. Ginter 


Problem Control System Briefing Charts 

CSM Open Problem List - dated April 23 , 1970 

CSM-108 Launch Readiness Report, issues 8 and 11 

24-Hour Notification Report 

Anomaly List - Apollo 12 

Apollo 13 REF (Friday night report) 

R&QA O 2 Tank Certification Report 

Mr. J. Whitten 




Functions and Organization Safety Office - MSCI II 38 .IB and 
Applicable Position Descriptions 
Power and Propulsion Division ( Fuel Cells/ Cryogenics) Applicable 
Position Descriptions 

MSC Safety Plan - M3CM 1701 (currently being revised) 

Boeing Apollo TIE System Safety Plan - D 2 -II 8036 -IA 
Organization Boeing System Safety 

Boeing System Safety Technical Management Review (typical) 
Functions and Organization ASPO - MSCI ll4l.l 
Assistant Program Manager Flight Safety, ASPO, MSCA 67-126 and 
Applicable Position Descriptions 
Requirements for Apollo Contractor Safety Plan (2- 17- 69 ) 

KSC-MSC Relationships at the Cape 

Apollo 13 Safety Support - Safety Office and Boeing System Safety 
Boeing - Selected Areas of Concern for Apollo 6 (Fuel Cells and 
Cryogenics) D2-118072-1 

Boeing - Apollo Spacecraft System Safety Assessments - Apollo 11 
and 13-2 volumes each 

Boeing - Apollo Spacecraft System Safety Assessment - Apollo 7 
to 13 Excerpts pertaining to CSM Cryogenic Oxygen System 
Detanking Cryo Oxygen Problem at KSC - DR-0512 
Nonmetallic Materials in the Oxygen Storage Tank Assembly 
Configuration Control Board Minutes regarding CSM Cryogenics 
MSC R&QA/Safety Suspect Flight Anomaly Report - Apollo 13 
Historical Summary of the Apollo 13 Cryo Oxygen Tank No. 2 
(S/N 10024 XTA000 8 ) 

MSC Safety Office Activities prior to and during the Apollo 13 
Incident through Splashdown, May 22, 1970 
Apollo 13 Pre- Launch R&QA Activities 
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RScQA (GE) Data Bank - Flight Anomalies 
Development Certification Test History of Cryo Tank No. 2 
Apollo FMEA (Cryo System); Single Point Failure Summary 
Sky lab FMEA (Cryo System) 

Apollo 204 Accident - Cryo Oxygen regarding Action Items (NASA) 
Apollo 20k Accident - Cryo Oxygen regarding Design Changes 
North American Rockwell Contract NAS9-150, System Safety Monthly 
Report, Period ending February 28, 1970 
System Safety Audit Report on North American Rockwell, Space 
Division, Downey, California, dated October 1, 1969 
North American Rockwell, Space Division, Safety Organization 
Fault Tree Analysis, Apollo 13 Incident, dated May 12, 1970 
Letter to NASA Hqs. , from M. L. Raines, dated April *9, 1970, 
Subject, R. Emerson Harris (Aerospace Safety Advisory Panel 
Report) "System Safety Review and Evaluation of the MSC and 
Its Principle Apollo Prime Contractors, period - October 7-l6, 
1968" 

Manpower Summary R&QA/Safety as of April 30, 1970 
Mission Log Book - Apollo 13 

Apollo Miss ion- Safety Concerns Status (Boeing) March 11,1970 
FRR Flight Safety Assessment, Apollo 13 Mission, dated February 26, 
1970 

Apollo 13 Spacecraft Readiness Status, dated March 6, 1970 
MSC Comments dated July 7, 1969, on Combined Aviation, Systems, 
and Industrial Safety Survey, MSC, April 14-17 , 19^9 
R&QA Assessment Statement Apollo 13 Flight Readiness Review 
R&QA Weekly Activity Reports to AA/R. R. Gilruth for periods for 
January 1, 1970 to May l6, 1970 inclusive 
RStQA Weekly Activity Reports to PA/OoI. McDivitt for periods 
January 1, 1970 to May l6, 1970 inclusive 
RScQA MDnthly Assessment Reports for October and November 19^9 
Phase II LM-7 CARR - Trip Report, April 16-17, 19^9 
CSM-109 FRR at KSC - Trip Report, January 16-17, 1970 
Mode IV Abort Report (Boeing), November 19 69 
CSM-109 Phase III CARR, Downey, June 3-4, 19^9 > Trip Report 
Apollo 13 R&QA Flight Readiness Assessment 

Documents Distributed to Board Members May 21, 1970 

Apollo Spacecraft Nbnmetallic Materials Requirements, MSC-PA- 
D-67-13, February 9, 1968, (Addendum No. 1, November 7# 19^9) 
Procedures and Requirements for the Flammability and Offgassing 
Evaluation of Manned Spacecraft Nonmetallic Materials, 

D-NA-0002, July 1968 

Nbnmetallic Materials Design Guidelines and Test Data Handbook, 
MSC-NA-D-68-1, Revision F, November 26, 1969 
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ENCLOSURE B 


The following reports were supplied to Apollo 13 Investigation Team 

members : 


1. "Apollo 13 Investigation, Panel 6 - Related Systems Evaluation 
Inputs from Reliability and Quality, Revision 1," NB/Vo-I'^-fO, dated 
May 15, 1970. 

2. "A Literature Survey of Reports of Unexpected Incidents in the 
Presence of Oxygen at Concentrations and Pressures Higher tnan that of 
the Normal Atmosphere," dated May 14, 1970, provided to Pane:l 8. 

3. "Spacecraft 109 - Shelf Incident Report," dated May 12, 1970, 
(copy attached). 



Enclosure B 
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Item 3 j 

NATIONAL AERONAUTICS ND SPACE ADMINISTRATION 
I MSC APOLLO 13 INVESTIGATION TEAM 


MANNED SPACECRAFT CENTER 
Houston. Texas 77058 


NUTT IU / . 

ATTN OF ND15/M70- l8 

MEMORANDUM TO : Leader, Apollo 13 Investigation Team 

FROM : Panel 9 Chairman 

SUBJECT : Spacecraft 109 - Shelf incident report 


The purpose of this report is to provide a chronological history of 
events leading to and including this incident at the contractor’s plant, 
Downey, California. 

On August 22, 1968, North American Rockwell (NR) Engineering 01x3 er (E0) 
M702138, was released, which required a change to vac-ion pumps to prevent 
corona discharge. The shelf on CSM 106 was selected to be removed and 
updated for CSM 109. This was a normal inline update. EO 702134-1 and 2 
authorized the rework and test of the 02 tank assembly and vac -ion pumps 
by Beech Aircraft Corporation. 

A manufacturing order (fair ticket) number 488757* was written August 24, 
1968, against EO M702138 to provide for removal and rework of the 0 2 
shelf V37-454200-41 s/C 106. 

On October 21, 1968, approximately 10:00 in the morning, service module 
106 was returned to test stand, position 2A, Bldg 290, NR Downey. The 
SM was rotated 90° from its normal installation position to accommodate 
the overhead crane, the 9EH-1275 (counter balance) and the 9EH-1009 
(0 2 shelf removal fixture). 

In this position, sector #4 of the SM would be facing the front of the 
stand. The floors of the stand are designed to fold back and the removal 
equipment can be easily attached. 

At approximately 3:30 p.m. on October 21, 1988, an attempt was made to 
remove the 0 2 shelf from SM 106. Hie 9EH-1009 shelf adapter model was 
attached to the 0 2 shelf and £he 9EH-1275 counter weight sling was attached 
to the overhead crane. The CT shelf was lifted approximately two inches, 
the counter balance was being adjusted, and the overhead crane was being 
mo.ed outward when the 9EH-1009 shelf adapter broke at the lower attachment 
weld. Immediately following the weld break on the 9EH-1009 shelf adapter, 
the 0 2 shelf dropped back to its mating surface in the SM structure. The 
9EH-1275 counter-weight sling swung up tting the test stand structure 
and damaging a light fixture. The area was immediately roped off and in- 
dustrial security was notified, lhere were no personal injuries. 
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Disposition record (DR) A80327, item #156, was initiated at the scene 
and photographs were taken of the c£ shelf, the immediate surrounding 
area and the broken 9EH-1009 adapter. After further investigation, it 
was discovered in preparing the CF shelf for removal from the service 
module, one of the 11 shelf retaining bolts was inadvertently left in 
place. As part of the corrective action on the disposition record #158, 
sequential inspections were performed on the shelf and in the surrounding 
areas. Dye penetrant inspections were made in the areas of the #3 radial 
beam stiffner and the structure was found acceptable. During the inspection 
of the surrounding areas, a small identation (ding) was noted directly 
above oxygen tank number two in the face sheet of the fuel cell shelf. 

This indentation was written up on disposition report A112436 #159 and 
later dispositioned as a normal standard repair. 

The 9EH-1009 fixture was repaired by adding stiffners to either side of 
the adapter fixture along with rewelding the tubing and the 0^ shelf was 
removed and taken to the systems test laboratory for the vac-ion modifi- 
cation and subsequent testing on October 22, 1968. 

A manufacturing order (no number) was issued October 28, 1968, to 
transport the 0^ shelf from Bldg 290 to Bldg 288 for further testing. 

On November 12, 1968, DR 140539 was written by the NASA Quality 
organization requesting NASA Systems Engineering evaluation for retest 
provisions resultant from the 0^ shelf drop incident. The retest re- 
quirements were evaluated and the necessary testing was accomplished during 
the time the 0^ shelf was in the test cell. 

The c£ shelf was returned to Bldg #290 on the 20th of November, 1968. 
Inspection for identification and damage was performed and the 0^ shelf 
was installed in SM 109 on the 22nd of November, 1968. 

The corrective action taken by NR is as follows: 

1. Industrial Security Report (B04453)» 

2. Corrective Action Record (U.C.) #A80327» 

3. W. H. Gray letter to D. D. Myers, NE/68L- 10-1373* dated October 23, 
1968, NR response 6GMA806T, dated November 4, 1968. 

4. HAMP Manual update to correct problem 10/29/70. 

5. Review and update of handling equipment. 

6. Review and revise existing planning. 

Enclosures 1 and 2 of this report are the photographs and documents 
assembled to describe this event. 

Copies have been provided to Structures and Mechanics Division and the 
Investigation Board to determine the impact of this drop and its relation- 
ship, if any, with the Apollo 13 incident. 



M. L. Raines 


Enclosures 
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